Skip to content

fix: Report watched file changes promptly with watchdog 7, and react to a touch - #577

Open
kinyoklion wants to merge 1 commit into
feat/overridesfrom
rlamb/overrides-python-windows-watch
Open

kinyoklion wants to merge 1 commit into
feat/overridesfrom
rlamb/overrides-python-windows-watch

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Requirements

  • I have added test coverage for new or changed functionality
  • I have followed the repository's pull request submission guidelines
  • I have validated my changes against all supported platform versions
  • Sync/async parity: matching changes made to async_* siblings (or N/A). N/A: the file watcher serves both clients.

Related issues

This resolves the SDK findings of the Windows stability assessment for the overrides work: plans/python-overrides-windows-test-stability.md in sdk-scratchpad.

Describe the solution you've provided

All three changes are in Watcher, in ldclient/impl/integrations/files/filedata.py, which the file-based override source uses in watching mode.

  1. watchdog 7 delays Windows events by a second, and starves them in a busy directory (Finding 2).
    • watchdog 7.0.0 (2026-10-31) changed its Windows reader. It now waits for the observer timeout after the first notification before delivering anything, and it delivers nothing while notifications keep arriving within that timeout.
    • With the default one-second timeout, a change to an override file was reported about a second late on Windows. It was not reported at all while another file in the same directory, such as a log, changed more often than once a second.
    • The assessment measured, on Windows Server 2025 with watchdog 7.1.0: a median latency of 1003 ms, and three of three changes not reported within 8 s next to a file appended every 200 ms. With a 100 ms timeout: 102 to 110 ms, and 3 of 3 reported.
    • The watcher now builds the observer with timeout=0.1. On Linux, the timeout only sets how often the idle inotify loop wakes.
    • Python 3.11 and later install watchdog 7 under watchdog>=3.0.0, so this affects any Windows deployment that selects ChangeDetection.WATCHING.
  2. A touch no longer reloads with watchdog 7 (Finding 3).
    • watchdog 7 reports a change to a file's metadata, such as a new modification time from os.utime, as an attrib event. Earlier versions reported it as modified.
    • The watcher ignored attrib, so touching a file to force a reload stopped working in watching mode, while it still works in polling mode.
    • attrib now counts as a change. A plain read does not produce it, so the reloads stay quiet. The reloader's content digest still skips applying an unchanged file.
  3. watchdog 6 on Windows can miss a change right after a watch starts (Finding 4).
    • watchdog 6.0.0, the last version that supports Python 3.10, returns from starting a watch after a fixed 10 ms. Its reader thread may not yet have asked for the first notification, and anything before that request is lost.
    • The assessment found that a deletion made immediately after the watcher was created was reported 0 of 10 times, and 10 of 10 times after one thread yield.
    • The watcher now yields once after each watch it sets up.

Tests in test_filedata.py:

  • a change is reported within 0.5 s;
  • a change is reported within 2 s while another file in the directory is appended to five times a second;
  • a change of the modification time alone is reported.

Describe alternatives you've considered

  • Pinning watchdog<7 in the test-filesource extra. Applications bring their own watchdog, so a pin would only hide the behavior from CI.
  • Running a slow safety poll under watching mode, so that a lost or starved notification is bounded. That is a design decision for the spec, not part of this fix.
  • Documenting that watching mode ignores a touch, instead of reacting to attrib. That would leave watching and polling behaving differently for the same file.

Additional context

Testing:

  • On Linux with watchdog 7.1.0, the touch test fails without this change and passes with it.
  • The latency and busy-directory tests pass on Linux either way, because inotify delivers each event at once. Windows CI runs them against watchdog 7, which they target.
  • Every watcher test passes with watchdog 6.0.0 and with watchdog 7.1.0. That includes the existing test that a read alone does not signal.
  • The file data and override source modules pass on watchdog 7.1.0 (263 tests).
  • The full suite passes (2,120 tests), without the Redis, DynamoDB and Consul store tests, which need servers that were not running.
  • mypy, isort and pycodestyle are clean.

This PR merges cleanly with #574 and #576.

Not in this PR, from Finding 6 of the assessment. None of these come from the overrides work:

  • a latency assertion in test_async_migrator.py that is too tight for Windows clocks;
  • a streaming retry test that is flaky on main;
  • the big segments status-polling contract test, which turns Linux CI on feat/overrides red in 4 of 5 runs and needs its own investigation before the merge into main.

The assessment also recommends reporting the watchdog 7 behavior upstream; that is not done yet.

Opened from termag


Note

Overview
Updates the file Watcher in watching mode so override/data reloads behave reliably with watchdog 6–7, especially on Windows.

The observer now uses a 100 ms delivery timeout instead of watchdog’s 1 s default, so changes are reported quickly and are not indefinitely delayed when another file in the same directory keeps notifying. attrib is treated as a content-relevant event so a mtime-only touch triggers a reload again under watchdog 7. After each directory watch is scheduled, a thread yield (time.sleep(0)) avoids missing an immediate post-start change on watchdog 6 for Windows.

New watcher tests cover sub-second latency, detection alongside a busy log file, and mtime-only updates.

Reviewed by Cursor Bugbot for commit 546a7f5. Bugbot is set up for automated code reviews on this repo. Configure here.

…to a touch

watchdog 7 on Windows holds notifications for the observer timeout, one second by default, and delivers none while another file in the directory changes more often. The watcher now uses a 100 ms timeout. watchdog 7 reports a touch as an attrib event, which now counts as a change. After scheduling a watch, the watcher yields so that the reader thread of watchdog 6 on Windows asks for notifications before the setup goes on.
@kinyoklion
kinyoklion requested a review from a team as a code owner November 11, 2026 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant